{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "hyperparamter_search.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "private_outputs": true,
      "collapsed_sections": [
        "Tce3stUlHN0L"
      ],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.7.0"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "Tce3stUlHN0L"
      },
      "source": [
        "##### Copyright 2019 The TensorFlow Authors.\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "tuOe1ymfHZPu",
        "colab": {}
      },
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MfBg1C5NB3X0"
      },
      "source": [
        "# Hyperparameter Search\n",
        "\n",
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/examples/blob/master/community/en/hyperparameter_search.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/tensorflow/examples/blob/master/community/en/hyperparameter_search.ipynb\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "xHxb-dlhMIzW"
      },
      "source": [
        "## Overview\n",
        "Hyperparamter tuning or search is somewhat of a black box, an art as it is so often referred to as is the process of choosing some of the parameters of a deep learning model in order to obtain the best possible performance for that architecture. There are quite a few tools out there that do a decent job of tuning parameters, but none are as straightforward, robust and state-of-the-art as Keras-Tuner. \n",
        "#  \n",
        "This notebook will show how the parameters can be tuned manually and using Keras-Tuner. But first, here's a peek at few of the tools:\n",
        "\n",
        "### HyperParameter Tuning search\n",
        "- `Hyperopt`: a popular Python library for optimizing over all sorts of complex\n",
        "search spaces (including real values such as the learning rate, or discrete values\n",
        "such as the number of layers).\n",
        "- `Hyperas, kopt or Talos`: optimizing hyperparameters for Keras model (the first\n",
        "two are based on Hyperopt).\n",
        "- `Scikit-Optimize (skopt)`: a general-purpose optimization library. The Bayes\n",
        "SearchCV class performs Bayesian optimization using an interface similar to Grid\n",
        "SearchCV .\n",
        "- `Spearmint`: a Bayesian optimization library.\n",
        "- `Sklearn-Deap`: a hyperparameter optimization library based on evolutionary\n",
        "algorithms, also with a GridSearchCV -like interface. [Link](https://github.com/rsteca/sklearn-deap)\n",
        "- `keras-tuner`: Bayesian as well as RandomSearch based tuning library that is known as \"Hypertuning for humans\"\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "MUXex9ctTuDB"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2Ks23ztk_eAK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "assert tf.__version__.startswith('2')\n",
        "\n",
        "print(f'{tf.__version__}')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VFEo5cXuNuCI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "from sklearn.metrics import accuracy_score\n",
        "from sklearn.model_selection import RandomizedSearchCV\n",
        "\n",
        "from tensorflow.keras.regularizers import l2\n",
        "from tensorflow.keras.layers import Dense, Dropout, Conv2D, Flatten, Activation\n",
        "from tensorflow.keras.wrappers.scikit_learn import KerasClassifier"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7gTyBUPIbhBz",
        "colab_type": "text"
      },
      "source": [
        "## Loading the datatset"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "7sGyQVklCifB",
        "colab": {}
      },
      "source": [
        "mnist = tf.keras.datasets.mnist\n",
        "(X_train, y_train), (X_test, y_test) = mnist.load_data()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "colab_type": "code",
        "id": "yIBDfFqMChFN",
        "colab": {}
      },
      "source": [
        "X_train = tf.cast(np.reshape(X_train, (X_train.shape[0], X_train.shape[1], X_train.shape[2], 1)), tf.float64)\n",
        "X_test = tf.cast(np.reshape(X_test, (X_test.shape[0], X_test.shape[1], X_test.shape[2], 1)), tf.float64)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "3FgkTUsQAUcX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "y_train = tf.keras.utils.to_categorical(y_train)\n",
        "y_test = tf.keras.utils.to_categorical(y_test)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9UF9AKHgajrh",
        "colab_type": "text"
      },
      "source": [
        "## Manual Hyperparameter Tuning"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dx_nROJfas9k",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = tf.keras.models.Sequential()\n",
        "model.add(Conv2D(32, (3,3), activation='relu', kernel_initializer='he_uniform', input_shape=(28,28,1)))\n",
        "model.add(Conv2D(64, (3,3), activation='relu', kernel_initializer='he_uniform'))\n",
        "model.add(Flatten())\n",
        "model.add(Dense(20))\n",
        "model.add(Dropout(0.2))\n",
        "model.add(Dense(10, activation='softmax'))\n",
        "model.compile(loss='categorical_crossentropy', optimizer=tf.keras.optimizers.Adam(0.001), metrics=['accuracy'])\n",
        "model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "L0cRUN9Fas7S",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model.fit(X_train, y_train, epochs=5, batch_size=128)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "PhdapEGqa4f4",
        "colab_type": "text"
      },
      "source": [
        "Although this works, there is an element of luck and expertise to tune hyperparameters effectively. The use of Keras-Tuner is discussed below that performs the tuning effectively."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "h1v5CRo6Zb4k",
        "colab_type": "text"
      },
      "source": [
        "## Keras-Tuner - Hyperparameter Tuning\n",
        "---\n",
        "### Features of Keras-Tuner\n",
        "- **Intuitive API**: As easy as 1,2,3\n",
        "- **State of the art hypertuner algorithms**\n",
        "- **Tunable architectures ready to go**\n",
        "- **Seamless experiments recording**: Automatic recording to analyse and reproduce your results\n",
        "\n",
        "**NOTE**: Do not download the Pypi version of keras-tuner. Follow the steps in the cell below for downloading."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "EpP6AmH8ZO9C",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# use pip install keras-tuner once https://github.com/keras-team/keras-tuner/issues/71 is fixed in the pip package\n",
        "!pip install -q git+https://github.com/keras-team/keras-tuner"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xOG0h4IKZO6x",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import kerastuner\n",
        "from kerastuner.tuners import RandomSearch"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VKnuKpg6CzxA",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Step 1: Wrap model in a function\n",
        "def model_fn(hp):\n",
        "    \n",
        "    # Step 2: Define the hyper-parameters\n",
        "    LR = hp.Choice('learning_rate', [0.001, 0.0005, 0.0001])\n",
        "    DROPOUT_RATE = hp.Float('dropout_rate', 0.0, 0.5, 5)\n",
        "    NUM_DIMS = hp.Int('num_dims',  8, 32, 8)\n",
        "    NUM_LAYERS = hp.Int('num_layers', 1, 3)\n",
        "    L2_NUM_FILTERS = hp.Int('l2_num_filters', 8, 64, 8)\n",
        "    L1_NUM_FILTERS = hp.Int('l1_num_filters', 8, 64, 8)\n",
        "        \n",
        "    # Step 3: Replace static values with hyper-parameters\n",
        "    model = tf.keras.models.Sequential()\n",
        "    model.add(Conv2D(L1_NUM_FILTERS, (3,3), activation='relu', kernel_initializer='he_uniform', input_shape=(28,28,1)))\n",
        "    model.add(Conv2D(L2_NUM_FILTERS, (3,3), activation='relu', kernel_initializer='he_uniform'))\n",
        "    model.add(Flatten())\n",
        "    for _ in range(NUM_LAYERS):\n",
        "        model.add(Dense(NUM_DIMS))\n",
        "        model.add(Dropout(DROPOUT_RATE))\n",
        "    model.add(Dense(10, activation='softmax'))\n",
        "    model.compile(loss='categorical_crossentropy', optimizer=tf.keras.optimizers.Adam(0.001), metrics=['accuracy'])\n",
        "    return model\n",
        "    "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "G9GdZ50QCzub",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner = RandomSearch(\n",
        "    model_fn,\n",
        "    objective='val_accuracy',\n",
        "    max_trials=5,\n",
        "    executions_per_trial=3,\n",
        "    directory='temp_dir')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "u3kr9AB8Czsk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner.search_space_summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "IPnHFy0gCzqN",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner.search(X_train, y_train, epochs=5, validation_data=(X_test, y_test))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "mrN7-zP2b0zz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "models = tuner.get_best_models(num_models=3)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vudn2Rjjb0w4",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "tuner.results_summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dyg4mM9lb6BZ",
        "colab_type": "text"
      },
      "source": [
        "## Upcoming Features in Keras-Tuner\n",
        "- **Ecosystem Intergration**: Integration out-of-the box with Colab, GCP and many more...\n",
        "- **Online Dashboard**: App to monitor the tuning on the go!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "UhNtHfuxCGVy"
      },
      "source": [
        "## References\n",
        "- Elie Bursztein: Cutting Edge Tensorflow at Google I/O 19 - [[Youtube Link](https://www.youtube.com/watch?v=Un0JDL3i5Hg&t=458s)]\n",
        "- Keras-Tuner GitHub Repository - [[GitHub Link](https://github.com/keras-team/keras-tuner)]"
      ]
    }
  ]
}
